NASA Contractor Report 178353 


COMBINED OPTIMIZATION OF IMAGE-GATHERING 
AND IMAGE-PROCESSING SYSTEMS 
FOR SCENE FEATURE DETECTION 


Nesim Halyo 
Robert F. Arduini 
Richard W, Samms 


INFORMATION & CONTROL SYSTEMS, INCORPORATED 

28 Research Drive 
Hampton, Virginia 


Contract NASi-17133 

AUGUST 1987 


,« 15 i-fll-1783«'l CCMBI8ED CII1BIZATI0H OS 
iun-SwMK *» IB4GE-FFCCESS1BG StSJEBS 

tea scam ee»tobe deiecticb Final 
r,fn:*r 1 i o s n B c nd u5/ t B F cl i s or tt,£l g3,3 5 


NASA 

National Aeronautics and 
Space Administration 


887-27991 

Onclas 

0097592 


Langley Research Center 

Hampton, Virginia 23665 



FOREWORD 


The work described in this report was performed by Information & Control Systems, 
Incorporated (ICS) under Contract Number NAS1-17133 for the National Aeronautics 
and Space Administration, Langley Research Center, Hampton, Virginia. Mr. Friedrich 
O. Huck was the NASA Technical Representative for this contract. Dr. Nesim Halyo 
directed the technical effort at ICS. 



TABLE OF CONTENTS 


FOREWORD i 

LIST OF FIGURES iv 


CHAPTER I. COMBINED OPTIMIZATION OF IMAGE-GATHERING 
OPTICS AND IMAGE-PROCESSING ALGORITHM FOR 


EDGE DETECTION 1 

1 . INTRODUCTION 2 


2. FORMULATION OF THE COMBINED OPTIMIZATION PROBLEM . 5 

5. OPTIMIZATION OF CHARACTERISTIC IMAGE PROCESSORS. . 9 

4. OPTIMAL IMAGE PROCESSOR FOR EDGE DETECTION .... 15 

5. OPTIMIZATION OF THE IMAGE GATHERING OPTICS AND 


SENSITIVITY CONSIDERATIONS 20 

6. CONCLUSIONS 24 

REFERENCES 26 


CHAPTER II. EFFECTS OF ELECTRICAL FILTERING ON IMAGE- 
GATHERING AND IMAGE-PROCESSING SYSTEM 


OPTIMIZATION FOR EDGE DETECTION 45 

1 . INTRODUCTION 46 

2. EFFECTIVE IMAGE-GATHERING SPATIAL RESPONSE .... 49 

3. OPTIMAL IMAGE PROCESSOR 56 

4. OPTIMIZATION OF THE IMAGE GATHERING SYSTEM .... 61 

5. CONCLUSIONS 64 

REFERENCES 66 


preceding PftSE 


r-.i A, 


m NOT 


Bft Cl / ( INTENTIONALLY BLANK 


iii 



LIST OF FIGURES 


CHAPTER Is 

FIGURE 1 . BASIC BLOCK DIAGRAM FOR PROBLEM FORMULATION .... 28 


FIGURE 2. SMOOTHING OF SMALL DETAIL BY CONVOLVING SCENE 
RADIANCE L(x) WITH A GAUSSIAN KERNEL, , OF 

STANDARD DEVIATION 0 29 

FIGURE 3. SENSOR (OTF) RESPONES 30 

FIGURE 4. SCENE POWER SPECTRA FOR VARIOUS LEVELS OF DETAIL, 

*»r 30 

FIGURE 5. EXAMPLE OF UNDERSAMPLING 51 

FIGURE 6. OPTIMAL EDGE DETECTOR USING V 2 32 

FIGURE 7. OPTIMAL EDGE DETECTOR USING V 2 Gp, p = .2 33 

FIGURE 8. OPTIMAL EDGE DETECTOR USING V 2 G fi ,f) = A 34 

FIGURE 9- OPTIMAL EDGE DETECTOR USING V 2 AND SQUARE 

APERTURE 35 

FIGURE 10. OPTIMAL EDGE DETECTOR TRANSFER FUNCTION 


PERSPECTIVE PLOT FOR V 2 S/N = 32, /i r = l,p e = .38 ... 36 

FIGURE 1 1 . OPTIMAL EDGE DETECTOR TRANSFER FUNCTION 

PERSPECTIVE PLOT FOR V 2 S/N = 32, p r = l,p e = .38 ... 36 
FIGURE 12. OPTIMAL EDGE DETECTOR TRANSFER FUNCTION 

PERSPECTIVE PLOT FOR V 2 S/N = 32, (i r = 1 37 

FIGURE 13. OPTIMAL EDGE DETECTOR POINT SPREAD FUNCTION. ... 38 
FIGURE 14. 1-D SIMULATIONS OF OPTIMAL EDGE DETECTORS FOR 

= . 2 


39 



LIST OF FIGURES (CONCLUDED) 


FIGURE 15. MEAN-SQUARED ERROR VERSUS OTF CUT-OFF FREQUENCY 

FOR V*G $ AND fi r = 3 40 

FIGURE 16. MEAN-SQUARED ERROR VERSUS OTF CUT-OFF FREQUENCY 

FOR V 2 Gp AND Hr = 1 41 

FIGURE 17. MEAN-SQUARED ERROR VERSUS OTF CUT-OFF FREQUENCY 

FOR SCENE RECONSTRUCTION Gp • LAND (i r = 3 42 

FIGURE 18. SENSITIVITY TO UNCERTAINTIES IN ACTUAL SCENE 

DETAIL, Pr . DESIGN BASED ON /x r = 3 43 

FIGURE 19. SENSITIVITY TO UNCERTAINTIES IN THE ACTUAL SIGNAL- 

TO-NOISE RATIO S/N; DESIGN BASED ON S/N OF 16. . .44 

CHAPTER II: 

FIGURE 1 . BASIC SCANNER BLOCK DIAGRAM 68 

FIGURE 2. EFFECTIVE IMAGE-GATHERING PSF, r(x,y) 69 

FIGURE 3. EFFECTIVE IMAGE-GATHERING OTF, f(v,w) 72 

FIGURE 4. IMAGE-GATHERING OPTICS 75 

FIGURE 5. LOW-PASS ELECTRICAL FILTER 77 

FIGURE 6. OPTIMAL EDGE DETECTOR OTF ( c .(x,y) = -V 2 L(x,y),p r = 3) . 78 

FIGURE 7. OPTIMAL EDGE DETECTOR PSF (c,(x,y) = - V 2 L(x,y), ft r = 3) . 81 


V 


CHAPTER I. 


COMBINED OPTIMIZATION OF IMAGE- GATHERING 
OPTICS AND IMAGE-PROCESSING ALGORITHM 
FOR EDGE DETECTION 


ABSTRACT 


This paper investigates the relationships between the image-gathering and image- 
processing systems for minimum mean-squared error estimation of scene characteristics. 
A stochastic optimization problem is formulated where the objective is to determine a spa- 
tial characteristic of the scene rather than a feature of the already blurred, sampled and 
noisy image data. An analytical solution for the optimal characteristic image processor 
is developed. The Wiener filter for the sampled image case is obtained as a special case, 
where the desired characteristic is scene restoration. Optimal edge detection is investigated 
using V 2 G as the desired characteristic, where G is a two-dimensional Gaussian distribu- 
tion function. It is shown that the optimal edge detector compensates for the blurring 
introduced by the image gathering optics, and notably, that it is not circularly symmetric. 
The lack of circular symmetry is largely due to the geometric effects of the sampling lattice 
used in image acquisition. The optimal image gathering OTF is also investigated and the 
results of a sensitivity analysis are shown. 


1. INTRODUCTION 


Much attention on edge detection has centered around the development of algorithms 
which process the sampled image data obtained by an image acquisition system [l] - [5]. 
Often these algorithms are finite difference approximations to linear isotropic differential 
operators, and are generally based on highly intuitive and heuristic arguments. 

On the other hand, a frequency domain filter which maximizes the energy within a 
specified interval about an edge (modeled as a step function) given the constraints that 
the filter response vanish at zero frequency and outside a specified frequency band and be 
symmetric is developed in [6], [7]. The result of this approach, namely the V 2 C? operator, 
where G represents a two-dimensional Gaussian distribution function, is also obtained in 
[2] which uses a heuristic argument to minimize a different objective function. 

A statistical approach to image reconstruction and edge detection using Wiener filter 
concepts is found in [8] - [13]. In general, the underlying approach in these studies is 
to develop image processing algorithms in order to determine a spatial characteristic of 
the image, such as its edges. In [12] and [13], the effects of the image gathering point 
spread function (PSF) are included for the Wiener filter case ([9], [10]) where the desired 
spatial characteristic is scene restoration. The relationship between the mean-squared 
error and the pupil function of the aperture at various signal-to-noise ratios is investigated 
for a class of one-dimensional pupil functions in [13]. In all cases, the effects of the image 
gathering sampling lattice on the optimal processor and the relationships between the 
image gathering and processing systems has not received much attention. In most studies, 
the effects of the image gathering PSF used in acquiring the image are not explicitly 
modeled and included in the development of the processing algorithms. 

In this paper, the objective is to determine a spatial characteristic or feature of the 
scene under observation, rather than a characteristic of the already blurred, sampled and 
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noisy image data. The image processing algorithm is designed to produce a spatial char- 
acteristic of the scene radiance observed from the samples of the image obtained by the 
acquisition system. In order to obtain the desired feature of the scene, the optimal image 
processor must compensate for the major effects of the image gathering system; namely, the 
blurring introduced by the image gathering optics, the aliasing effects due to the sampling 
lattice used, and the sensor noise effects, or more generally, image acquisition errors. 

When the objective is formulated as a characteristic of the scene, rather than the im- 
age, both the image gathering system and the image processing algorithm have significant 
effects on the error in estimating the scene characteristic, as shown in Figure 1. Thus, 
both the image gathering and image processing systems must be optimized to obtain the 
total imaging system; i.e., the optimal image gathering and processing systems must be 
compatible with each other. 

In Section 2, a combined stochastic optimization problem is formulated where the 
objective is to estimate a spatial characteristic of the scene. The optimization criterion 
considered is the spatially-averaged mean-squared error between the desired scene char- 
acteristic and the output of the image processing system. Characteristics which can be 
obtained from the scene using a linear operator are considered. This class of characteristics 
includes scene reconstruction, low-pass, band-pass or other filtering tasks, estimation of 
the gradient, V 2 G, etc. 

An analytical expression for the optimal characteristic image processor in terms of 
the image gathering system parameters is obtained in Section 3. This general expression 
explicitly shows the relationships between the optimal image gathering and processing sys- 
tems for an arbitrary desired characteristic. The Wiener filter for the sampled image case 
is obtained as a special case of the optimal characteristic image processor, where the de- 
sired characteristic is selected to be least-mean-square scene reconstruction. The optimal 
characteristic image processor for the special case of a scene characteristic corresponding 
to the DOG (Difference-Of-Gaussian) operator is also used in [17] to investigate the rela- 
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tionships between the criteria of fidelity and information density. Some general properties 
of the optimal characteristic image processor are mentioned. 

Section 4 investigates the optimal image processor for edge detection using scene char- 
acteristics corresponding to V 2 G operators. It is shown that the optimal edge processor 
tries to compensate for the blurring introduced by the image gathering optics by appropri- 
ately amplifying the spatial frequencies which have been attenuated in image acquisition. 
In particular, it is shown that the optimal edge detector is not circularly symmetric even 
when the desired characteristic has a circularly symmetric spectrum. This is shown to be 
due to the non-symmetric effects of the sampling lattice used in image acquisition. 

The combined optimization of the image gathering optics and characteristic image 
processor, as well as the sensitivity of the processor is investigated in Section 5. General 
expressions for the spatially-averaged mean-squared error in estimating an arbitrary scene 
characteristic are presented. The optimization of the image gathering optics for edge 
detection is investigated both according to the mean-squared error criterion and according 
to the robustness or insensitivity of the image processor. 

In this paper, the specific results have largely centered around edge detection and the 
V 2 G characteristics, as the edges of a scene are generally accepted to contain much of the 
important information in an image. However, the results of Sections 3 and 4 are general 
and applicable to the estimation of a large class of characteristics; so that the methodology 
of combined optimization of the image gathering and processing systems can be applied 
to the analysis and synthesis of a variety of imaging problems. 
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2. FORMULATION OF THE COMBINED OPTIMIZATION PROBLEM 


Consider an image gathering system whose output is fed to an image processor to 
obtain the total imaging system output, as shown in Figure 1 . The image gathering 
system is described by the Optical Transfer Function (OTF) of the optics, the sampling 
lattice and an additive noise process. Thus, the image gathering system transforms the 
scene into a sampled image which is blurred and corrupted by noise. 

The sampled image data is then processed by an algorithm with the objective of 
obtaining some spatial characteristic or feature of the scene under observation. As the 
objective is to determine a feature of the object scene under observation, rather than 
a feature of the already blurred, sampled and noisy image, the image processor must 
appropriately compensate for the blurring introduced by the OTF and for the particular 
sampling lattice, while accounting for system noise. In this formulation, it is clear that 
both the parameters of the image gathering system and the image processing algorithm 
must be optimized in order to best achieve the objective of obtaining the desired scene 
characteristic. 

For notational simplicity, x,u and k will be used to represent column vectors with 
two components (xi,X2) T » ( w i» w 2) T > and (ki ,k 2 ) T , respectively . 1 The product kuf of two 
column vectors will represent the vector {kiUi,k2U>2) T • The product of a row vector by a 
column vector (e.g., u> T x = w jXi +W2X2) will represent the usual inner (or dot) product. 
In general, we will use x as the spatial location variable and u as the spatial frequency 
variable. 

As shown in Figure 1 , let L(x) represent the scene radiance, f(w) 2 the OTF of the 
image gathering optics, corresponding to the Point Spread Function (PSF) r(x). Let 

x The superscript “T” denotes the transpose of a vector or matrix. 

2 The superscript denotes the two-dimensional Fourier Transform. 
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x(fc) be the sampling points ( Xi(fci),X 2 (fc 2 )) defining the sampling lattice used, m(k) the 
noisy and blurred image at the sampling points x(k), and n(&) the equivalent noise after 
sampling. We consider the particular scene radiance L(x) viewed, and the particular noise 
sequence n(k) encountered to be realizations of second order stationary random processes. 
In this way, the imaging system is designed for a class of scenes and noise patterns having 
the same Power Spectral Density (PSD or Wiener spectrum). The scene L{x) and the 
noise n(k) are assumed to be statistically independent processes. We will denote the 
power spectral densities of the scene radiance, L(x), by ^r,(w), while the PSD’s of other 
random processes will have appropriate subscripts. 

With the notation described, the basic equations defining the image gathering system 
can be formally expressed as 


L(x) = / e i2 ™ Tx L{u)du, xeR 2 (l) 

Jr 3 

s(x) = [ t(x — x') L(x') dx' , (2) 

Jr 3 

s(x) = [ e i2 ™ Tx f(u)L(u>)du> , ( 3 ) 

Jr 3 

M(x) = s(x) + n(x) , (4a) 

m[k) = M(xk) = s(x(k)) + n(k) , — oo < ki,k 2 < oo . (4b) 


For ease of notation, a single integral sign over R 2 (Euclidian 2-space) is used to 
denote a double integral. It should be noted that Fourier theory for second-order random 
processes requires the use of stochastic Riemann-Stieltjes integrals [18], [14]. However, 
to maintain the familiarity of deterministic Fourier transform relationships, (l), (2) and 
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(3) are formally expressed in the same manner as deterministic functions, but should be 
interpreted as stochastic integrals described in the Appendix of [14] for mathematical rigor. 

As shown in (4b), the image m(k) is a sequence of points obtained from sampling 
the continuous image M (x) according to the selected sampling strategy. Thus, the image 
processor must operate on this image sequence in order to best estimate the desired scene 
characteristic, c(x). In this paper, the sampling intervals are normalized to unity, unless 
stated otherwise. We will consider the class of image processors shown below. 

OO 

r (x) = ^ t p (x — x(k)) m(k) , xeR 2 (5) 

k~ — oo 

Thus, the processed output is (in general) an infinite sum of the samples of the image 
weighted according to the image processor PSF, r p (x). The processing algorithm produces 
an output at any point x, not just at the sampling points x(k). Note that when the output 
is restricted to the sampling points, 1 (5) becomes a discrete convolution corresponding 
to a spatially-invariant linear system. However, in the general case, this processor does 
not satisfy all the requirements for a spatially-invariant system. For the class of image 
processors selected, the processing algorithm is completely determined by the PSF r p (x) 
or equivalently by its Fourier transform, f p (w). 

The class of scene characteristics considered in this paper is the class that can be 
obtained from scene, L(x), using a linear, spatially-invariant operator. Thus, if the desired 
scene characteristic is c(x), then 


c 



t c (x — x') L(x') dx' 
e %ixu T x L(u>) du . 


The Wiener spectrum of the desired characteristic can be found to be 
3 Other selections are also possible. 


( 6 ) 

( 7 ) 
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<M W ) = l^cHI 2 ^i(w) , 


oj e R 2 . 


( 8 ) 


The class of scene characteristics considered is quite large, as any linear and spatially- 
invariant operator where the characteristic, c(x), has a finite variance, can be optimized. 
The class includes operators such as the Gaussian class, the Difference of Gaussian (DOG) 
class, V 2 G, V, high-pass, low-pass and band-pass filters, etc. 

The criterion of optimization used in this paper is the spatially-averaged mean-squared 
error. While other criteria often produce useful results [14], [15], the mean-squared error (or 
derivatives such as fidelity) provides a simple, easy to understand criterion which produces 
analytical and insightful solutions. The criterion of optimization can be expressed as 

J = 717 / E \ c ( x ) ~ r ( x )\ 2 dx > ( 9 ) 

|A|-*oo \A\ J A 

where E denotes the statistical expectation operator and |A| the area of the surface A. 

The combined optimization problem can now be posed as the problem of determining 
the image gathering system OTF, f (w), and the characteristic image processing algorithm 
determined by t p (x) or f p (u;) which minimize the spatially-averaged mean-squared error, 
J, between the desired scene characteristic, c(x), and the imaging system output, r(x), 
subject to constraints on f(u>) and f p (u). 
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3. OPTIMIZATION OF CHARACTERISTIC IMAGE PROCESSORS 


We will approach the combined optimization problem in two steps. The first step is 
to optimize the characteristic image processor, f p (w), as if the OTF f(u;) were a known, 
fixed function. This results in the optimal characteristic image processor as a function of 
the image gathering optical transfer function, f (w). Then, with this optimal characteristic 
image processor, the criterion J can be minimized to obtain the optimal OTF, f (w), within 
a class or realizable optics, which constitutes the second step. 

In this section, the first step will be solved analytically. This important result pro- 
vides the relationships between the optimal characteristic image processor and the image 
gathering system. The results provide insight by pointing out the parameters which are 
more important than others. 

When the parameters of the image gathering system are fixed, the optimal processor, 
f p (u>), operating on the image data is the processor whose output, r(x), is the least mean- 
square estimate of the scene characteristic, c(x). It is well-known that, for second-order 
random processes, the optimal output, r(x), is the projection of the scene characteristic, 
c(x), onto the Hilbert Space generated by the image data {m[k) t — oo < k < oo}, (e.g., see 
[19]). Thus, applying the Projection Theorem to the first step of our optimization problem, 
it can be shown that a linear characteristic image processor minimizes the mean-squared 
error E\c{x) — r(x)| 2 if, and only if, 

£(r(x) m(k)) = E(c(x) m(k )) , Vk . (10) 

Furthermore, such r(x) exists and is unique with probability 1. 

Now to determine the characteristic image processor satisfying (10), first note that 
from (2) and (3), 
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( 11 ) 


m(k) = M(x(k)) = f e’ Wx(fc) f(u) L(u) du + n(k) . 

Jr 3 

Combining (11) and (7), 

E(c{ x) m(k)) = f e *2»rw T (*-*(*)) fc ( w ) f *( w ) £ L ( W ) dw (12) 

J R* 

where we have used the facts that m(k) is real and that the scene radiance, L(x), and the 
noise, n(k), are uncorrelated. 

To obtain the left hand side of (1), first note that 


m(k) = M[x{k)) = / e‘ W *( fc ) M{u) dw 

Jr* 

(13) 

= f e *' 2 »(“' T +JwJ )z( fc ) M(u) du> 




= [ m{u - ju 0 ) du 

(14) 


R 3 


where u 0 is the Nyquist frequency vector and j is an arbitrary integer. Taking the Fourier 
transform of (5) 

r(u) = fp(u>) rh(w) , u e R 2 (15) 

where rh(cj) is the discrete Fourier transform of the sampled image data, i.e., 

OO 

m(u) = ^ m{k) e -»2«-w T *(*) , z(fc) = kX . (16) 

fc = — OO 

On the other hand, using Theorem 3, in the Appendix of [14], 


r 



gt‘2iru» T 



1 f p(w) t(u) du 
e i2 ™ Tx T p {u)M(uj - 


j u 0 ) du . 


(17) 

(18) 
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Combining (14) and (18), we obtain the LHS of (10). 


E(r[x) m(fc)) = Y' f e i2 ™ T ( x x ( k » t p (w)4> M (u - ju 0 )<bj 

Jr* 


( 19 ) 


j=-oo 


= / c *'2*-w T (*-*(*)) f p ( w ) $ m (u) du) 

Jr » 


( 20 ) 


Substituting (20) and (12) into (10), we obtain the expression for the optimal charac- 
teristic image processor 


f p (u;) = f c (w) 




where $ m (w) is given by (e.g., [12], [13]) 


weR 2 


( 21 ) 


OO 

ImH = ^2 “ ku} o )\ 2 4>l{u ~ ku 0 ) + 4> n {u) 

fc=— oo 

— ^a(w) + ^n(w) . (22) 

Thus, an analytic expression for the optimal characteristic image processor transfer 
function in terms of the parameters of the image gathering system has been obtained. The 
optimal characteristic image processor PSF, r p (x), used in (5) is obtained simply as the 
inverse Fourier transform of the optimal transfer function, f p (w) , given in (21). 

The expression obtained, (21), can be viewed as a compatibility measure between the 
characteristic image processor and the image gathering system, as this expression explic- 
itly points out the relationships between the two system parameters which produce the 
smallest mean-squared error in the estimation of a scene characteristic. Whether or not 
the imag e gathering system is the “best” system to determine a particular scene charac- 
teristic, the relationship shown in (21) holds as the least-mean-squared error characteristic 
image processor for that case. While this relationship will be investigated for particular 
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classes of scenes, optical transfer functions, etc., some important general properties of the 
optimal characteristic image processor are worthy of note. 

The optimal characteristic image processor is a linear function of the desired scene 
characteristic. As an example of the additive property, note that the optimal image pro- 
cessor for the characteristic V 2 (•) = (•) + -i-r (•) (r c (w) = — (w 2 + u> 2 ) = — p 2 ), is the 

OX 2 X 2 

sum of the optimal image processors for the characteristics gp- (•) (r c i(w) = — w 2 ) and 
(') {jc2 (w) = — w 2 ). This linearity property carries further into linear operators. Sup- 
pose that c p (x) is the scene characteristic corresponding to the characteristic transfer 
function t Cg (u>), and that we would like to estimate a new characteristic, c(x), defined by 
the convolution 


c(x) = a(x) * c 0 (x) = [a(x) * r Co (x)] * L(x) (23) 

c(u?) = a(w) c 0 (w) = (a(w) f Co (w)] L(u) . (24) 

Then, the optimal characteristic image processor for c(x) is given by 

f p (w) = a(u>) r po (w) , (24a) 

r p (x) = a(x) * r po (x) , (24b) 

r(x) = a(x) * r 0 (x) , (24c) 


where r po (x) and r 0 (x) are the optimal image processor and its output for the characteristic, 
c 0 (x). 

4 The symbol “* n denotes the two-dimensional convolution integral when applied as a binary 
operator. 
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An important special case of the optimal characteristic image processor is the Wiener 
filter [8], [9], [10], for sampled images. The Wiener filter corresponds to the case where 
the desired characteristic is simply the scene radiance; i.e., 

c(x) = L(x) , f e (u>) = 1. (25) 

Substituting (25) into the general expression (21), results in the least-mean-squared error 
estimator of L, or the Wiener filter for sampled images. 

= ^ . ««** (26) 
^m(w) 

The Wiener and edge filters developed here were used in [17] to investigate the relation- 
ships between the criteria of information density and fidelity for scene reconstruction from 
sampled images. 

It should also be noted that the optimal characteristic image processor given in (21) 
has a commutative property. In cases where it is desirable to perform some processing 
locally, this produces a flexibility in implementation and analysis. For this purpose, write 
t p (u) in (21) as the product 


f p (w) = a(w) S(w) = S(w) a(w) , (27a) 

t p (x ) = a{x) * b(x) = b(x) * o(x) . (27b) 

Then the optimal r(x) can be implemented in either of the following ways, provided the 
terms in brackets are finite. 5 


5 This commuting property cannot be extended past the sampling process which does not 
commute in the usual manner. 
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r(x) = a(x) * 


(28a) 


= 6(x) * 



6(x — x(k)) m(k) 
a[x — x(A:)) m{k) 


= X) r p(* ”*(*)) m ( fc ) 

OO 


(28b) 

(28c) 


: (w) = a(w) S(a;) rh(uj) = f p (u) m(<v) 


(29) 


Finally, it should be noted that the output r(x) is the optimal estimate of the scene 
characteristic, c(x), for every point x, i.e., i?|r(x) — c(x)| 2 is minimized for every xeR 2 . 
Therefore, the optimal characteristic image processor, defined by (21) and (5), can be 
used to reconstruct c(x) as a continuous function, or to optimally estimate c(x) over a finer 
grid than the sampling lattice (or the same lattice) as the image data. Applications of 
the optimal image processor in this fashion include optimal interpolation between sampling 
points, or obtaining estimates over a finer grid which can be fed to a digital high-resolution 
display device for reconstruction. If the basis reconstruction function of the display device 
is known (e.g., Gaussian, cubic interpolation, etc.), a preprocessor may sometimes be 
desirable. 
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4. OPTIMAL IMAGE PROCESSOR FOR EDGE DETECTION 


The optimal characteristic image processor given by (21) can be used to estimate any 
scene characteristic that can be expressed in the form of (6) or (7); i.e., any characteristic 
that can be obtained from the scene with a linear operator. An important characteristic of a 
scene is the location of its edges, or sudden, sharp changes of intensity in the scene radiance, 
L(x). To investigate the implications of the optimal characteristic image processor to the 
task of edge detection, we will restrict attention to specific classes of scene and noise 
power spectral density image gathering system OTF’s, and characteristics which contain 
edge information. 

For this purpose, we will use the V 2 Gp operator, as the zero-crossings of this operator 
are known to contain information on edges [3], [7]; V 2 denotes the Laplacian operator and 
Gp denotes a Gaussian exponential with a standard deviation of /?, viewed as a PSF. In 
this context, the standard deviation, /?, of the Gaussian PSF determines the level of detail 
desired. Figure 2 shows a one-dimensional convolution of a scene containing a variety of 
edges with a Gaussian at various values of the standard deviation. As can be observed, 
the sharp edges are smoothed, and edges which occur close to each other (i.e., the high 
level of detail) are blurred increasingly, as /? increases. Small detail smoothed in this 
manner becomes distinguishable from the image gathering system noise, and aliasing error. 
Variation of (3 allows us to specify the level of detail desired in estimating the location of 
edges. For example, if we want to accurately determine the finest detail, /? can be set to 
a small value. On the other hand, if the smallest level detail is not necessary or desirable 
for a given application, (3 can be set to a higher value. Thus, the characteristic transfer 
function used is of the form 6 

®Note that the r c (w) expression used corresponds to —V 2 Gp. 
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1 


(30a) 


f c ( W ) = p 2 e -^' a / 2 


p 2 = uf + 


c(x) = -V 2 {Gp{x) * L(x )> . (30b) 

It should be noted that a subpixel accuracy is inherent in our formulation. Whether 
we are interested in small or large detail, the accuracy with which edges are located is 
an important property of an edge detector. As the system output shown in (5) can be 
computed for as fine a grid as desired, the zero-crossings of the optimal estimate of V 2 Gp 
can be determined to an arbitrary accuracy within the pixel in which they occur. Figure 
14 shows one-dimensional simulations of the optimal edge detector estimating the charac- 
teristic c(x) = —V 2 Gp * L( i), for a Gaussian smoothing factor /? = .2. As the zero-crossings 
of the output, r(x), with appropriate thresholding provide estimates of the edge locations, 
it may be noted from Figure 14 that the error in locating the edges of L(x) is significantly 
less than a pixel. Also note that due to the Gaussian smoothing, “small edges” are not 
detected, which is the intent in introducing the smoothing. 

The class of image gathering system OTF’s, f (u>), which will be considered here have 
the form 


f (u/) = t-?!* , p 2 = w\+wl. (31) 

Figure 3 shows a cross-section of these circularly symmetric OTF’s. This class of OTF’s, 
while not exhaustive of all realizable transfer functions, is representative of the basic char- 
acter of many apertures, and electro-optical device transfer functions. This form belongs 
to the more inclusive class of OTF’s described in [20]. The case of a square aperture OTF 
is also considered for completeness. 

It is important to note that for purposes of comparison and optimization, the OTF’s 
Me normalized so as to have equal “effective” aperture areas. Thus, for each OTF consid- 
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ered, the volume under the PSF of the image gathering system is kept constant. 

The class of scene radiances considered here will be modeled as a homogeneous and 
isotropic random field; in particular, we use a wide-sense stationary second-order random 
process with power spectral density 4>l(w) and covariance function ^l(z) as shown below. 


4>l{w) = 


2fl~M? o\ 

[1 + 4n 2 Mr ( w i + w|] 3 / 2 


(32) 


4> L (x) = E(L(x')Hx' + x)) = ol , r 1 = xl+xl. (33) 

Figure 4 shows cross-sections of the scene spectra in (32) for various values of (x r . 

The PSD in (32) corresponds to a random set of two-dimensional pulses whose width 
obeys an exponential probability law with average or mean width of Mr, and whose mag- 
nitude obeys a Gaussian or Normal probability law with zero mean and variance a\ [21], 
[22]. This interpretation provides a physical significance to the scene statistics which is 
highly appropriate for edge detection investigations. Furthermore, the parameter Mr at- 
tains a meaning directly applicable to the problem of optimal edge detection. Since fi r is 
the average distance between two edges, it is a meaningful description of the amount of 
detail in the scene. For example, a Mr value of 1 would imply that, on average, the scene 
contains an edge at every sampling interval (normalized to unity); so that the distance 
between some edges would be smaller and others larger than the sampling interval. 

Suppose that, as shown in Figure 5, the scene contains four edges between two sam- 
pling points. From the samples of this scene shown in Figure 5, it is not possible to deter- 
mine the location and magnitude of all four edges in the scene no matter what method of 
edge detection is used. This would be a situation occurring regularly throughout the scene 
for the case of fi r = 1/4. Thus, for edge detection purposes, the sampling interval selected 
would be very large, and must be reduced considerably if we are to reliably find the edges 
in the scene. Hence, we will consider optimal designs for the more realistic sampling rates 
according to the level of detail corresponding to Mr values greater than 1. 
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Figures 6, 7, 8 and 9 show cross-sections of the optimal characteristic image processor 
(21) for edge detection using the characteristic V 2 Gp * L for various levels of desired detail, 
/ 3 , for various levels of detail present in the scene, fi rt and for various image gathering OTF 
cut-off frequencies, p c , as well as the case of a square aperture, all using a square sampling 
lattice. The optimal edge detector displays certain broad characteristics in all the cases 
shown despite the extent of variations in the parameters. 

In all cases, the optimal edge detector vanishes at zero frequency, peaks near but 
before the Nyquist frequency (at .5), and attenuates at high frequencies. The optimal 
transfer functions tend to peak in the vicinity of .4 for a Nyquist frequency of .5 for almost 
all cases. However, the magnitude of the peaks varies considerably, largely according to 
the image gathering optics used in acquiring the image, but also with other variables. 
Thus, the optimal edge detector attempts to reduce the blurring introduced during image 
acquisition by amplifying the attenuated frequencies till close to the Nyquist frequency, 
where the aliasing and noise levels start to dominate. 

Figures 10 and 11 show the two-dimensional perspectives of the optimal edge detector 
for the case of square and hexagonal sampling. Figure 12 shows the perspective plot for 
a square aperture with a square sampling lattice. It is clear that the optimal character- 
istic image processor for edge detection is not circularly symmetric even when the scene 
statistics, the image gathering optics and desired characteristic all have circular symmetry. 
The optimal edge detector is seen to be highly dependent on the sampling lattice used in 
the image gathering process. Thus, the optimal edge detector attempts to account for all 
three major effects introduced in an image acquisition: blurring, noise and sampling. 

Figure 13 shows the PSF, r p (i), corresponding to the optimal edge detector using 
square sampling. This PSF is used as shown in (5) to implement the optimal characteristic 
image processor. The square shape of the optimal PSF is evident in the plots. This PSF 
can be used to estimate the desired characteristic as a continuous function, or at a much 
finer grid than the image to achieve subpixel accuracy as shown in the one-dimensional 
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simulations in Figure 14. It should be noted that the square shape of optimal edge detector, 
is a rather natural consequence of the sampling lattice effects in compensating for the cross- 
correlation of the signal at the sampling points. Also note from Figure 13c that the basic 
shape of the optimal PSF varies relatively little as the sensor cut-off frequency moves 
displaying some insensitivity to p c . 
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5. OPTIMIZATION OF THE IMAGE GATHERING OPTICS 
AND SENSITIVITY CONSIDERATIONS 


As discussed in Section 3, the combined optimization problem is divided into two 
steps: the optimization of the characteristic image processor and the optimization of the 
image gathering Optical Transfer Function (OTF). Having solved the former analytically 
as a function of the OTF, we use this expression (21) to evaluate the spatially-averaged 
mean-squared error, J, and minimize it over the class of OTF’s selected. 

We first obtain a general expression for the mean-squared error where the image 
processor f p (u>) is not necessarily the optimal processor. After some manipulation of (18), 
(3) and (4a) , it can be shown that 


(x) = / j 2 ™ T *\ V e i2 * JW ° x f p (u> + ju 0 ) 

Jr 3 Lrr'L 


j = -oo 


[f (w) L(io) + iV(w)] dw (34) 


where N(u>) is the continuous Fourier transform of the noise n(x). From this expression, it 
becomes clear that the system output, r(x), is not usually a wide-sense stationary process 
over R 2 . However, its statistics are periodic, so that when spatially averaged, the variations 
vanish. Also note that the sampled sequences {r(x(A:) -f A)} are stationary, due to the 
periodic nature of r(x). 

Substituting (34) and (7) into (9), and after considerable manipulation, the general 
expression for the spatially-averaged mean-squared error can be shown to be 


J= f {|*c(w)| 2 $£,(“) -2Re[f*(u) r(u/) f p (w)] 

J JZ a 

+ \r P (u)\ 2 <£rn (<*>)} du (35) 

where f p (ui) is an arbitrary image processor rather than the optimal one, and 4> m {u>) is 
the PSD of the sampled image which includes aliasing, as given in (22). Therefore, this 
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expression can be used to evaluate the mean-squared error in estimating a characteristic, 
c(x), using an OTF, f(w), and an arbitrary image processor, f P M- 

When the optimal characteristic image processor in (21) is used, the mean-squared 
error becomes 



x ! kHI 2 feM 

^a(w) + $»(u;) 
1 

1 + 


\t c {u )\ 2 dxj 


$ c (w) duj = f 2 h ^ 4> c {u) dw 

Jr* 


(36a) 

(36b) 


= ~ f I'M r c (w)| 2 du , (36c) 

Jr* 

where ^a(w) is the PSD of the aliasing error and 4> m {yJ) the PSD of the discrete noise, and 
<r 2 is the variance of the desired characteristic, c(x); i.e., 


°l - E |c(i)| 2 = / |f c ( w )| 2 fo( u ) du . (37) 

Jr* 

These expressions show the relationships between the mean-squared error and the signal- 
to-noise ratio, S/N, and the information density, h(u>). 

It should be noted from both (36) and (35), that when the image gathering OTF, 
f (w), vanishes or is small over a frequency band, it is essentially not possible to recover the 
desired characteristic over that band. Thus, if the desired characteristic has high-frequency 
content, an image gathering OTF which acquires the high-frequency content even at the 
expense of some aliasing will result in a lower mean-squared error. Of course, to the extent 
possible, it is desirable to select a sampling interval so that most of the characteristic to 
be estimated is contained within the Nyquist band. 

Figures 15, 16 and 17 show the spatially-averaged mean-squared error when using the 
optimal characteristic image processor for the characteristics V 2 G^ * L and simply L for 
various levels of noise; i.e., for optimal edge detection and for optimal restoration of the 
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image. As can be seen from the plots, the OTF shape which minimizes the mean-squared 
error depends on the frequency content of the desired scene characteristic. As the desired 
characteristic is smoothed by the Gaussian operator Gp, which shifts the frequency content 
lower, a shift to lower cut-off frequency can be noticed. However, the basic shape of the 
curves in most cases is such that a cut-off frequency in the interval from .5 to .7 produces 
little variation from the optimal. Further note that for low signal-to-noise ratios, the trend 
is to open the OTF band-width. In the case of edge detection, where the high-frequency 
or fine detail is to be estimated, the tendency is to prefer a higher cut-off frequency, p e , 
for the image gathering OTF. While in the case of image restoration, where the desired 
characteristic has relatively low-frequency content, the mean-squared error is optimized at 
a lower cut-off frequency. 

An important characteristic of any design is its robustness; i.e., its lack of sensitivity 
to variations in parameters about the scene. In particular, the optimal characteristic image 
processor depends on our expectation of the scene, namely, \i r and <f> n - Now suppose that 

A 

we design the image processor based on values /i r p>^np> hut the actual scene corresponds 
to different values say fi r and 4> n . It is important to know how much loss from the optimal 
design for the actual case ( p r ,4>n ) we suffer by incorrectly designing for p. rp and <£ np . 
Figures 18 and 19 show the loss from the optimal when the design is based on fi rp = 3, 
and a signal-to-noise ratio of 16. Figure 18 shows the change, or loss, of the mean-squared 
error for edge detectors for different values of mean spatial details in the actually observed 
scene, i.e., n r . It is seen that, when the actual scene detail is 3 times larger or smaller than 
expected, an OTF design with a cut-off frequency, p c , higher than .6 produces minimal 
loss of performance. Note that, at these levels of mismatch, the sensitivity is low for all 
cut-off frequencies shown. Also note that, if the actual detail is larger than expected, the 
loss of accuracy is low. When the actual level of detail is much smaller than expected 
by the design (recall that the sampling interval is set to a value of 1), the edge detection 
sensitivity is accordingly higher, with some preference for pc s in the vicinity of .6. 
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Figure 19 shows the change, or loss, in mean-squared error when the signal-to-noise 
ratio is mismatched. When the actual signal-to-noise is lower than expected (i.e., higher 
than expected noise level), the curves are relatively flat over the range where the mean- 
squared error itself is small (see Figures 15 and 16). When the actual signal-to-noise ratio 
is larger than expected (i.e., lower than expected noise level), OTF cut-off frequencies 
higher than .6 appear to be preferable. Thus, designing for a noise level that is high and 
using OTF cut-off frequencies higher than .6 produce favorable noise robustness. 

Interestingly, this sensitivity analysis tends to favor cut-off frequencies unless the 
expectations about the scene are grossly misjudged as represented by the curve for /z r = 
1/3. The noise sensitivity curves indicate that, for higher p c , designing for high noise can 
be advantageous. 

Finally, the general expressions (35), (36) for the spatially-averaged mean-squared 
error in estimating a scene characteristic obtained in this section provide a general method 
of evaluating the combined image gathering and image processing systems, as well as 
providing insight into the basic tendencies and trade-offs involved in designing complex 
imaging systems. 
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6. CONCLUSIONS 


This paper formulates a combined optimization problem where the optical transfer 
function (OTF) of the image gathering system, the noise level, the sampling lattice as well 
as the image processing algorithm are all design parameters to be selected according to 
the optimization criterion chosen to be the spatially-averaged mean-squared error. In this 
formulation, the objective is to determine a spatial characteristic of the scene radiance 
rather than a characteristic of the already blurred, noisy and sampled image. 

With this formulation of the objective, the relationships between the image gathering 
system parameters and the image processing algorithm are clearly defined. To estimate a 
characteristic of the scene, the image processor must account for, or try to eliminate, any 
distortion that has been introduced in the image acquisition process. 

The optimization of the image processor for a given image gathering system to estimate 
a scene characteristic is achieved analytically, which is the main result of the paper. An 
analytical expression for the optimal characteristic image processor in terms of the image 
gathering optical transfer function, the signal-to-noise ratio, and the sampling lattice effects 
is presented. This expression shows the relationships between the optimal characteristic 
image processor and the parameters of the image gathering system explicitly. The results 
are valid for characteristics which can be obtained from the scene by linear transformations. 
The Wiener filter for the case of sampled images is presented as a special case of the optimal 
characteristic image processor developed. 

The optimal processor for edge detection is investigated by selecting scene charac- 
teristics from the form V 2 G. The main results obtained show that the optimal edge 
detector 1) is not circularly symmetric but is highly dependent on the sampling lattice, 
as it attempts to account for its effects, 2) tends to peak near the Nyquist frequency, and 
vanish at zero and high frequencies, 3) attempts to reduce the blurring, aliasing and noise 
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effects introduced during image acquisition and 4) appears most insensitive to expected 
scene variations for p e values between .7 and .9, corresponding to a 3 dB cut-off near the 
Nyquist frequency. One-dimensional simulations of the optimal edge detector demonstrate 
subpixel accuracy in determining the location of scene edges. 

Expressions for the spatially-averaged mean-squared error between the desired scene 
characteristic and its estimate are obtained in terms of the scene power spectral density, 
the image gathering optical transfer function, the characteristic image processor and the 
desired scene characteristic. These expressions provide some insight into the behavior of 
the error for various designs. The results indicate that the optimal image gathering OTF 
depends on the frequency content of the scene characteristic desired. The basic tendency 
displayed is to increase the bandwidth of the image gathering system when the desired 
scene characteristic has high-frequency content. 

Finally, the formulation presented provides an objective method of analyzing a com- 
plex imaging system where the objective is to determine a scene characteristic. 
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FIGURE 1. Basic block diagram for problem formulation. 
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FIGURE 8. Optimal edge detector using V 
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FIGURE 11. Optimal edge detector transfer function 
perspective plot for S/N = 32, V* r = 
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FIGURE 13. Optimal edge detector point spread function. 
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FIGURE 18. Sens: 



CHAPTER II. 


EFFECTS OF ELECTRICAL FILTERING ON 
IMAGE-GATHERING AND IMAGE-PROCESSING 
SYSTEM OPTIMIZATION FOR EDGE DETECTION 


ABSTRACT 

This paper investigates the effects of phase distortion due to on-line electrical filtering 
in the image-gathering system, and extends the results of Chapter I on the combined 
optimization of image-gathering and image-processing in estimating a scene characteristic 
or feature. An effective image-gathering point-spread function (PSF) including the effects 
of both electrical and optical systems is developed. It is shown that the phase distortion 
appears in the effective image-gathering PSF as a shift in the location of the PSF maximum, 
an asymmetry about the maximum and a decaying “tail” which effectively widens the 
sensor’s footprint. The edges of the acquired image are not only blurred, but also shifted 
in position along the scan direction. This effect further stresses the importance of the 
combined image gathering and processing approach. It is shown that the optimal image 
processor compensates for both phase and magnitude distortion in the image to the extent 
compatible with noise attenuation, and continues to display a lack of circular symmetry 
while being highly influenced by the sampling lattice used in image acquisition. Finally, it 
is shown that the spatially averaged mean-squared error and its sensitivity to variations in 
the scene detail and noise statistics are both independent of the image-gathering system 
phase characteristics, when the optimal image processor is used to correct magnitude and 
phase distortion. 
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1. INTRODUCTION 


The performance of line-scan imaging systems has received considerable attention in 
the electro-optical system design literature [1] - [12]. Line-scan imaging systems, such 
as TV cameras and optical-mechanical scanners generally use on-line electrical filtering 
to shape the image-gathering system characteristics along the scan direction. The usual 
considerations in selecting the electrical filtering to be performed involve the trade-off 
between the reduction of noise and aliasing effects by filtering out high frequencies and the 
blurring introduced by the attenuation of high frequency detail. 

Whereas the effects of blurring due to the attenuation of certain frequency bands, or 
transfer function magnitude effects, have been investigated (e.g., [12], [13]), the effects of 
blurring due to the phase distortion introduced by the electrical transfer function have 
received little attention in the literature. 

Due to the symmetry in the aperture of most designs, optical system transfer functions 
are real functions and have zero phase, except for optical aberrations. On the other hand, 
on-line electrical filter transfer functions are complex functions and have non-zero phase. 
The transfer function of the overall image-gathering system thus becomes complex with 
non-zero phase characteristics, and results in a phase distortion of the image. 

In this paper, the effects of electrical filter phase distortion on the image-gathering 
system transfer function, the image and the optimal image processor are investigated. The 
methodology of combined optimization of image-gathering and image-processing used in 
this paper follows the same lines as the one introduced in [14]. The results of [14] are 
extended to image-gathering systems which have non-zero phase. 

As shown in Figure 1, the image-gathering system consists of the optical transfer 
function, f 0 (u, oj) , which usually has zero phase, the electrical filter transfer function, 
h«(w / ), where u/ is the temporal frequency variable while u and u represent the spatial 
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frequency variables, the sampling process and additive noise or, more generally, sensor 
errors. The image-gathering system is followed by the image processor, f p (t/,w), which 
produces the overall system output, r(x, y). 

The objective is to determine a spatial characteristic or feature of the scene rather 
than a characteristic of the image data which contains the effects of blurring due to both 
magnitude and phase distortion and has been contaminated by noise. The class of spatial 
characteristics considered here consists of characteristics which can be obtained from the 
observed radiance field by a linear operator. The image processor uses the sampled image 
data to estimate the desired scene characteristic. The optimal image processor must com- 
pensate for the magnitude and phase distortion introduced by the image-gathering system, 
the aliasing effects introduced by the sampling lattice and the sensor noise. 

In Section 2, the effects of the electrical filter or, more generally, the effects of temporal 
processing, are modeled in terms of spatial processing characteristics. Combined with 
the optical transfer function, this results in an overall image-gathering transfer function 
which is complex, therefore producing both magnitude and phase distortion in the image 
acquired. This effective point-spread-function (PSF) of the image gathering system is 
shown to be shifted away from the origin and to have a “tail” along the same axis. The 
effective PSF shows that the edges in the image will be shifted relative to the location 
of the edges in the scene, due to the phase distortion. It is also seen that the tail of the 
effective PSF will produce a further blurring effect. 

In Section 3, the optimal image processor for image-gathering systems with non-zero 
phase is investigated. It is seen that, in order to eliminate (minimize) phase distortion, 
the optimal image processor must have exactly the opposite phase as the image-gathering 
PSF. The magnitude characteristics of the optimal image processor are seen to be the same 
as for image-gathering systems with no phase distortion. In particular, it is seen that as in 
[14], the optimal edge detector is not circularly symmetric due to the effect of the sampling 
lattice used in image acquisition. 
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In Section 4, the spatially averaged mean-squared difference between the scene charac- 
teristic desired and the system output is considered. It is shown that the spatially averaged 
mean-squared error is independent of the phase characteristics of the image-gathering scene 
when the optimal image processor is used, irrespective of the scene characteristic or fea- 
ture being estimated. It is also shown that the sensitivity of the optimal image-gathering 
system to variation (or uncertainties) in the scene and noise statistics is also independent 
of the phase characteristics of the image gathering system, as long as the optimal image 
processor is used. Therefore, the magnitude characteristics of the optimal image-gathering 
system obtained in [14] remain valid for non-zero phase image acquisition, as long as the 
optimal image processor is used. 
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2. EFFECTIVE IMAGE- GATHERING SPATIAL RESPONSE 


To model the effects of electrical filtering (or processing) on the overall electro-optical 
system, we will consider a scanner where the scanning axis, say y, is perpendicular to the 
x-axis in which the relative motion of the scanner center of mass and the scene occurs. 
Other types of scanning strategies can be analyzed and modeled in a similar fashion, but 
will not be treated here. As shown in Figure 1, the electrical filtering is actually performed 
on the analog signal which is obtained by the optical system. Thus, the scanning axis, 
y, runs along the time axis. We will assume that the scanner’s relative motion along the 
x-axis is much slower in comparison to the scanning motion of the optical axis. In other 
words, the scanning speed is assumed to be much faster than the relative motion of the 
device. 

Let h e (t) be the impulse response of the electrical filter processing the signal obtained 
by the Optical Transfer Function (OTF), f 0 (u,u>). Thus, the Electrical Transfer Function 
(ETF), h e (u , ) t is given by the usual one-dimensional Fourier transform 

*.(<»')= r , a) 

J — oo 

where w # is the temporal frequency variable in units of cycles per sec. In most continuous 
scan systems, the electrical processing used is low-pass filtering with the ETF correspond- 
ing to standard low-pass filters such as the Butterworth, Tchebycheff, Bessell filters. 

The input-output relationship of the electrical filter is given by the one-dimensional 
convolution 


o(t) = f h e {t') i(t — t r ) dt' 
J — OO 


( 2 ) 
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where »(<) and o{t) are the input and output signals, respectively. It is assumed here that 
transient effects due to initial conditions have died down to negligible levels. 

On the other hand, when the optical axis points along (x,y), the optical system input- 
output relationship is given by the two-dimensional convolution 

g(x,y) = f f r 0 (z', y') L(x — x\y — y') dx' dy' (3) 

J— oo J —oo 

where L(x>y) is the scene radiance input to the sensor and y(x,y) is the output, and 
r„(x,y) is the optical Point Spread Function (PSF) corresponding to the OTF, t 0 (u,u). 

Now suppose that, at time t, the optical axis points along (x(t),y(i)) producing the 
signal y(x(t),y(i)). Then, the analog output of the electrical filter is given by 

s{x{t),y(t))= f h e (t')g(x(t- t'),y{t- t'))dt' , (4) 

J — OO 

where the output of the electronics, s(x(t),y(t)), has been associated with the optical axis 
direction (x(f),y(t)) rather than simply time, t. 

As mentioned earlier, the scanning motion along y is much faster than the motion 
along x. Thus, we assume that y(i — t') varies along the scan direction, while x(t — t') 
remains essentially unchanged over the period of a scan; i.e., 

x(t — t') « x(t) , (5) 

y(t - t‘) = y(t - t ') = yt — yt ’ , (6) 

when t — t* is of the order of a scan period, but is much larger than the electrical filter 
time constants, and where y is the linear scan rate. Substituting (5) and (6) into (4) and 
manipulating, 
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(7) 


«(*(<)> y(0) = f MO 9 {x(t) t yt - yt') dt ' 

J — oo 

= \ S- oo he (t) " y/ ) dy/ 


= / f T t {x\y')g{x{t) - x\y{t) - y')dx'dy' (8) 

*/ — OO J —OO 

where 

y' = y*' , (9) 

M*'»y') = » ( 10 ) 

and 6(x') is the Dirac delta function. 

It is seen that the effect of the electrical filter in the spatial domain can be expressed 
by the PSF r e (x / , y 1 ) and a two dimensional convolution, as shown in (8). 

Taking the 2-D Fourier transform of (8), and using (3), 

s(i/,w) = t e (i/,u>) g(i/,u>) = f e (v,w) f 0 (i/,w) L(v, w) . (11) 

Taking the Fourier transform of (10), and manipulating, 

f e (t/,u;) = h e {yu) = f e (w) , (12) 

where u> is the spatial frequency variable corresponding to the scan direction; i.e., the 
y-axis. It should be noted that since f e varies only with u> and is independent of u> we will 
treat it as a function of u> alone to emphasize this fact. 

From (11) and (12), it is seen that the effective OTF for the combined electro-optical 
system is given by 
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= t e (u) f 0 ( u,u) = h e (yu) f 0 {u, w) , (13) 

where h t is the temporal transfer function of the electrical filter, and y is the scan rate. 
Thus, given h e (u>') and the scan rate, it is a simple matter to obtain the effective spatial 
OTF. 

S imil arly, it can be shown that the effective system PSF, r(a;,y), can be obtained as 
a one-dimensional convolution. 

r(x,y) = f \h e (^\ r 0 (x,y- y')dy' , (14) 

j — oo y \ y ) 

which is the inverse Fourier transform of the effective system OTF, f (x/,u;). 

It is important to note that any temporal effects which can be expressed in terms of 
a convolution can be analyzed in this manner. For example, thermal dynamics which may 
be present in the sensors can be included in the term h e (u>') and analyzed in the same way 
as electrical processing. 

The most si gnifi cant aspect of (13) is the fact that the effective OTF, is now 

a complex-valued function. Whereas the optical transfer function, f 0 (u,uj) t is usually a 
real-valued function, the electrical transfer function, h e (u/), is rarely, if at all, real-valued. 
Thus, in general, t e (u) and f(u, u>) now have magnitude as well as phase characteristics 
due to a non-zero imaginary part. 

The existence of non-zero and non-linear phase characteristics in the image-gathering 
system essentially produces a “blurring” effect which adds to the blurring due to the 
magnitude characteristics. Any known optical phase aberrations can also be included in 
?„(!/, w). 

The tr ansf er functions involved can be expressed in terms of their magnitude and 
phase in the form: 


he(w') = , r 0 (i/,u;) = | t 0 {v,u)\e^M . (15) 


52 



The effective system OTF can now be written as 

t(v,u;) = |he(ywOl|ro(^w)|c‘ [ ^ Ww)+ ^ ( * / ' u,)1 (16) 

From (15), it is seen that, when the optical system has no phase aberrations, the effec- 
tive system OTF phase is precisely the electronic phase, while the magnitude effects are 
multiplicative and depend on both optical and electrical system characteristics. 

It should be noted that for causal, or non-anticipating, electrical filters, the impulse 
response is constrained by 


h e (t) = 0 , t < 0 (17a) 

so that the filter can be implemented in real time. This causality condition simply requires 
that the output at time t depend only on the input till time t so that the current output of 
the filter does not depend on the future input. From (10), this causality condition requires 
that the effective PSF of the electrical filter satisfy the condition 

r«(z,y) = o , y < 0 . (17b) 

Figure 2 shows the effective image-gathering PSF, r(x, y), while Figure 3 shows the 
corresponding OTF; i.e., the effective image-gathering system OTF including both optical 
and electrical filtering effects. The optical system PSF and OTF used in the analysis 
are shown in Figure 4. These correspond to a square with sides of 1° convolved with a 
Gaussian kernel to model the edge effects for the obstructed portion of the field-of-view. 
Figure 5 shows the magnitude and phase characteristics of a 2 nd order Butterworth filter 
which is the class of electrical filters used in this study to illustrate the phase distortion 
effects. 

The main effects of electrical filtering on the image-gathering process are most clearly 
seen in the effective PSF of the image-gathering system shown in Figure 2. Comparison of 
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the optical system PSF (Figure 4) before the electrical filter and the effective PSF which 
includes the effects of both the optics and electronics, indicates that the effective PSF is 
shifted along the scan axis, y, is no longer symmetrical and displays a “tail” . The basic 
square shape of the field of view is seen to have become more rectangular since the electrical 
filtering occurs along the scan axis which corresponds to the time axis. The response in 
the cross-scan x-axis is essentially unchanged. This results in a widening of the effective 
field of view along the scan direction. 

The shift of the effective image-gathering PSF implies that the sensor output has a lag, 
or a distorted delay; i.e., the sensor output is obtained not only from the incoming radiation 
at the present time, but by appropriate weighting of past values of the incoming radiation. 
In spatial terms, this means that the sensor output, hence the image, is shifted relative to 
the scene in a distorted manner. This shift and the asymmetry of the PSF are due solely 
to the presence of phase effects; the blurring due to the magnitude characteristics is an 
additional effect. It should be noted that the shift associated with different frequencies 
is not uniform or constant; so that high frequency features and low frequency features 
are shifted by different amounts, producing the “distorted shift”. Linear phase versus 
frequency characteristics would produce a constant shift for all frequencies which could 
easily be accounted for in post-processing. However, on-line or causal filters cannot produce 
linear phase characteristics, but may approximate linearity over a given frequency band. 

The effect of different cutoff frequencies, u> e , corresponding to the half-power, or 3 db, 
point can be seen by comparing Figures 2a, 2b and 2c. It is seen that the lower the cutoff 
frequency, the larger the shift and the more pronounced the tail. It is generally known 
that the highest cutoff frequency consistent with the goal of reducing aliasing and noise 
would usually produce the best results. The phase distortion effects shown in Figure 2 
further suggest that the sensor aperture be designed with consideration to the electrical 
filter in order to achieve the desired effective image-gathering point-spread function. 

The shift in the effective image-gathering PSF shows that the edges of the image data 
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will be shifted relative to scene edges along the scan direction. This is a further indication of 
the necessity for the combined image-gathering and image-processing approach presented 
in [14]. An edge detector which does not make use of the effective image-gathering PSF 
will at best locate the scene edge with an error equal to the shift introduced during image- 
gathering. Whereas an edge detector which makes use of the effective image-gathering 
PSF can reduce the errors in edge location by appropriate processing as will be discussed 
in the following sections. 
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3. OPTIMAL IMAGE PROCESSOR 


Prom the previous section, it is clear that depending on the accuracy required (de- 
sired) and the frequency content of the signal, the effects of electrical processing are not 
necessarily negligible. The sensor design must trade off noise and aliasing versus phase dis- 
tortion. Also, the processing performed on the image data obtained by the sensor should 
try to reduce these effects according to the frequency content of the feature to be extracted 
from the image data. 

A useful criterion for evaluating the level of accuracy in estimating a feature is the 
spatially averaged mean-squared error. This criterion can be expressed in the form 

J = ^lirn^ pj- J j E\c(x,y) - r(x,y)| 2 dxdy (19) 

where E denotes the statistical expectation operator, |A| the area of the surface A, c(x,y) 
the characteristic or feature of the scene to be estimated and r(x, y) the estimate of the 
desired feature. 

The system shown in Figure 1 can be described by the equations 

s(x t y)= f f r(x',y')L(x- x',y- y')dx' dy' , (20) 

J — oo J — oo 

m )k = s(xj,y k ) + n jk , - oo < j , k < oo , (21) 

where the effective system PSF, r(x, y), satisfies (14) - (16). 

We restrict the system output, r(x,y), to be of the form 

OO 

r ( x >y)= X) r p (x - xy,y - y k )m jk , x,yeR (22) 

j,k=~oo 
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The system output, r(x, y), which must estimate the desired characteristic of the scene 
is a weighted infinite sum of the sampled, noisy measurement, my*. The particular image 
processor is determined by the choice of the weighting function, r p (x,y). 

The class of scene characteristics or features that will be considered here is the class 
that can be obtained from the scene radiance, L(x, y), using a linear, spatially invariant 
operator. Thus, 

c ( a: »y)=/ [ r c {x',y')L{x - x' ,y - y')dx' dy' , (23) 

J — OO J — oo 

where r e (x,y) is the PSF corresponding to the linear operator. Note that the Wiener 
spectrum, ^ c (i/,w), of the desired scene characteristic is given by 

tcfaw) = |f c (i/,w) | 2 &,(*/, w) , (24) 

where <^£,(i/,u;) is the Wiener spectrum or power spectral density (PSD) of the scene 
radiance, considered to be a second-order stationary random process. The noise ny* will 
be assumed to be a second-order stationary random sequence, statistically independent of 
the scene, L{x,y). 

In [14], it is shown that the image processor, f p (u,uj), which minimizes the spatially 
averaged mean-squared error, J in (19), under the constraints (20) - (23) is unique and 
given by 




(25) 


where the superscript denotes the complex conjugate and is given by [12] 


^m(&sw) = 4>»(w) + la^w) + 4>n(v,u) > 


(26) 




(27a) 
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(27b) 


oo 

4>a{v,u) = 53 i v o,U- ku 0 ) 

(j,*)^(0,0) 

i/ 0 and u> 0 are the Nyquist frequencies along the u and u> axes, respectively, 4> a (i/, w) is the 
PSD of the aliasing error, 4> n (v,u>) is the discrete PSD of the noise sequence {ny* , — oo < 
j , k < oo}. 

The effects of non-zero phase in the effective image-gathering OTF are due to the 
complex conjugate term in (25). The magnitude characteristics of the optimal image 
processor remain essentially unchanged. In particular, note that 


If r L - Ml - l r «( |/ » w )ll r (*'»") I M u >“) 

- j — 7 : 


i/,u)eR 


(28a) 


V?p(^w) = <pc{v,w ) - 

= <p c (*/,w) - <pe(yu) - , v,ueR (28b) 

where ^> p (i/,u 7 ), and <p(v,u>) represent the phase functions of f p (i/,u>), f c (i/,u>) and 

respectively. 

Figure 6 shows the magnitude of the OTF of the optimal image processor for edge 
detection. The image-gathering system used is the smoothed square aperture followed by 
2 nd order low-pass Butterworth filters at various cutoff frequencies, w c , discussed in the 
preceding section. The sampling strategy used is a square lattice with sampling interval 
normalized to unity in both the x- and y-axes. The feature or characteristic estimated is 
— V 2 * L ; i.e., the negative Laplacian of the incoming radiance, which is assumed to be a 
sample function from a second-order stationary random process with PSD 


4>l(v,u) 


2 TT/xfoi 

[1 + 27T/* r (i/ 2 + W 2 )] 3 A 


(29) 
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It is seen that this class of radiance inputs has circular symmetry. The parameter \i r 
corresponds to the average distance between two edges, and represents a measure of the 
level of detail present in the scene. 

As can be seen from Figure 6, the optimal edge detector transfer function, f p (u,u), 
vanishes at zero frequency, peaks near but before the Nyquist frequency at 0.5, and atten- 
uates high frequencies. The major characteristics of the optimal edge detector magnitude 
are essentially the same as shown in [14] where the image gathering OTF, f(i/,w), is as- 
sumed to have zero phase. The only essential difference in the case considered in this 
paper, is that the optimal edge detector has non-zero phase. Thus, in this case, the op- 
timal characteristic image processor tries to eliminate the phase shift introduced by the 
image gathering optics and electronics, as well as trying to reduce the magnitude induced 
blurring. 

From the perspective plots in Figure 6, it is clear that the optimal edge detector for this 
case is not circularly symmetric, but is mainly influenced by the sampling lattice. Thus, 
the optimal edge detector attempts to reduce all three of the major effects introduced 
during image acquisition: blurring, noise and sampling. In this case, blurring has two 
components: magnitude induced and phase induced. The optimal edge detector attempts 
to minimi ze both types of blurring. 

The similarity of the magnitude characteristics for the non-zero phase case seen here 
and the zero phase cue discussed in [14] is quite striking. Nevertheless, the effect of the 
Butterworth filter magnitude characteristics can be seen by comparing the different cutoff 
frequency plots. The effect is seen most clearly at the low cutoff frequency of 0.3, where 
the peaks along the scan axis (w) are lower than those along the cross-scan (u) axis. This 
demonstrates that while the sampling lattice effects are the most pronounced ones, the 
optimal image processor is also influenced by non-symmetric effects in the effective image- 
gathering system. Figure 7 shows the optimal PSF for estimating — V 2 L(x, y) from the 
sampled image. The effect of the non-zero phase of the optimal edge detector is clearly 
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seen in these plots. Note that the optimal PSF is shifted in the negative scan- or y-axis; i.e., 
the shift in the optimal PSF is in the opposite direction from the shift in the effective image- 
gathering PSF (see Figure 2). It is clear that the optimal image processor is correcting the 
distorted shift present in the image. Further note that the optimal image processor PSF 
displays a rectangular character more clearly noticeable at low cutoff frequencies, where 
the effective image-gathering PSF has a rectangular character. 

The main strategy of the optimal image processor is seen to be one of subtracting 
surrounding samples from the center sample with appropriate weighting, which is an ap- 
proximation to the Laplacian operator. The rectangular character and the distorted shift 
combine to account for the rectangular character of the image-gathering PSF. However, 
the main characteristics of the optimal image processor are quite similar to the one shown 
in [14] (Figure 14) where the image-gathering system has zero phase distortion. 
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4. OPTIMIZATION OF THE IMAGE- GATHERING SYSTEM 


In the preceding section, it has been shown that the optimal image processor for 
estimating a scene characteristic accounts for any known phase shifts introduced by the 
image gathering system. Thus, when an electrical filter having non-zero phase is used, 
the image processor itself has non-zero phase, as evidenced by (28), in an attempt to 
eliminate this phase distortion. Therefore, the optimal image processor depends on the 
phase characteristics of the effective image-gathering OTF. 

The spatially averaged mean-squared error, J, defined by (19) is known to be [14] 


=/:/: 


lf c (t/,fa;)l 2 ^(t/,u;) 

OO j — OO J ^ |f(t/,o;)l 2 ^L(^^) 


-duduj 


(30) 


4>a{v,u) + 4>n(v,u) 

when the optimal characteristic image processor in (25), or equivalently the expression in 
(28), is used. 

A 

From this expression and the expression for the aliasing error spectrum, <^ 0 (i/,u;), in 
(27), it is clear that the spatially averaged mean-squared error, J, depends only on the 
magnitude characteristics of the image-gathering OTF, f (i/,w). Alternately, the error is 
independent of the phase characteristics of the image-gathering system when the optimal 
image processor for the desired feature is used. It is important to note that when the op- 
timal image processor is not used, the phase characteristics of the image-gathering system 
may influence the error according to the particular suboptimal image processor used. 

The optimal characteristic image processor f p (i/, u>) depends on the scene’s statistical 
characteristics as described by and the effective discrete noise PSD, 4> n (i/,uj). In 

most cases, the true statistics of the scene and noise are not accurately known, so that 
the optimal image processor is based on uncertain parameters in 4>h{y> cj) and 
Therefore, the robustness, or lack of sensitivity, of the optimal image processor to errors in 
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parameters such as level of scene detail, /x r , and signal-to-noise ratio, S/N, is an integral 
part of the design process. 

It is of interest to see whether the robustness of the optimal characteristic image 
processor is affected by the phase characteristics of the image gathering system. To analyze 
the sensitivity, let be an arbitrary image processor; then the mean-squared error 

can be shown to be [14] 


J, = [ f [ l^c (&S <*>) \ 2 $l [y* w) 

J — oo J — oo L 

- 2R e (f*(i/,o;)f(x/,a;)fp(i/,a;)) fa,{u y u)) 


+ |f'(i/,w)| 2 ^ m (i/,w) 


dudoj 


Let 4 >' l (is, uf) and be the PSD’s used in designing f'(t/,w), and let 


O'. w ) = u>)t * (u, u) , v, weR , 

where ^(i/.u;) is given by 


(31) 


(32) 


OO 


\ f ( U ~ M.W - k(Jo)\ 4 >' L {v - jv 0 ,U 

j,k=-oo 

+ ^n( |/ » w ) 

Substituting (32) into (31), and manipulating, 


— ku) 0 ) 


(33) 


J ' = / / [|f.0'.w)| , $i(»',<<>) - |r c (l/,w)| J |f(l/,w)| 2 

J -OO J —oo L w) 

2^l(«',^A Wdw (34) 

Observation of (34), (33), (26) and (27) reveals that J' depends only on the magnitude 
of the image gathering system, and is independent of its phase characteristics. Further 
note that since J uses the optimal image processor while J' does not, it follows that 
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A J = J' - J > 0 


(35) 


Thus, the change in the mean-squared error due to using and $' m (i /,o>) in 

A A 

designing the image processor, when the true quantities are w) and is given 

by A J in (35) which, of necessity, is non-negative. From the previous argument, it is clear 
that J, J # , hence A J are independent of the phase characteristics of the image gather- 
ing system. This result might have been expected since the suboptimal image processor 
considered uses the correct phase characteristics for the image-gathering system PSF and 
uses the optimal processor expression in (25), albeit with incorrect statistics for the radi- 
ance signal and noise. In fact, observation of the general error expression in (31) indicates 
that if the phase of the suboptimal image processor is exactly the negative of the effective 
image-gathering OTF, then A J will be independent of the phase function. 

It is seen that both the spatially averaged mean-squared error, and the robustness 
properties of the optimal image-gathering system depend only on the magnitude of the 
image gathering system. 
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6. CONCLUSIONS 


This paper extends the results of [14] on the combined optimization of the image- 
gathering and image-processing systems to the case where the image-gathering system 
produces phase distortion in the acquired image due to electrical filtering. 

The effects of electrical filtering or, more generally, temporal processing such as ther- 
mal dynamics, is transformed into an equivalent spatial filter operating on spatial inputs. 
When combined with the optical transfer function, the overall effect of image-gathering 
optics and electronics can be described by a complex transfer function containing both 
magnitude and (non-zero) phase characteristics. Thus, the acquired image contains both 
magnitude and phase distortion before being sampled according to the sampling lattice 
and being contaminated by noise. 

The effective point spread function of the electro-optical image-gathering system is 
characterized by a shift in the PSF maximum away from the origin in the scan direction 
and by a “tail” asymptotically decaying to zero in the scan direction, as shown in Figure 2. 
As evidenced by this effective PSF, the effect of electrical filtering significantly modifies the 
imag e-gathering characteristics. Sharp edges in the observed scene are not only blurred, 
but also shifted in position along the scan direction. The amount of magnitude and phase 
distortion in the acquired image depends on the (spatial) frequency content of the incoming 
radiance. 

The optimal image processor which minimizes the spatially averaged mean-squared 
error for any linear scene feature is shown to eliminate the phase shift introduced during 
image acquisition. The optimal image processor itself has non-zero phase for this pur- 
pose, its phase equaling the negative of the image-gathering system phase. The optimal 
image processor for edge detection is not circularly symmetric, but is highly dependent 
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on the sampling lattice, and attempts to minimize the blurring, aliasing and noise errors 
introduced during image acquisition. 

It is also shown that, when the optimal image processor for the desired characteristic is 
used, the spatially averaged mean-squared error is independent of the phase characteristics 
of the image-gathering system and depends solely on the magnitude of the image-gathering 
transfer function. Similarly, the robustness characteristics are also shown to be indepen- 
dent of the phase characteristics of the image-gathering system; i.e., the change in the 
spatially averaged mean-squared error due to errors in the scene level of detail or noise 
statistics is solely dependent on the magnitude of the image-gathering transfer function, 
when the optimal image processor is used. 

These results imply that when the optimal image processor is to be used, the basic 
trends obtained in [14] are valid for non-zero phase image acquisition. Thus, in the design 
of the image-gathering system, the results indicate that if the image-gathering system is 
designed for l) lower signal-to-noise ratio as well as 2) higher scene detail than generally 
expected, and 3) with a 3 dB cutoff near, but lower than, the Nyquist frequency, it is 
possible to achieve nearly optimal mean-squared error. This is possible even if the actual 
signal-to-noise ratio turns out to be higher or the scene actually observed has a lower 
level of detail than used in the design, as long as the optimal image processor is used to 
eliminate the phase distortion introduced in image acquisition. 

If the optimal image processor cannot be used due to computational restrictions or 
other constraints, the image processor output may have errors depending on both magni- 
tude and phase characteristics. In such cases, the errors can be analyzed using the effective 
image-gathering PSF developed in Section 2 and the general expression for the spatially 
averaged mean-squared error given in (31). 
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FIGURE 2. EFFECTIVE IMAGE-GATHERING PSF, T(x,y) 





FIGURE 2. EFFECTIVE IMAGE-GATHERING PSF, x(x,y) 
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FIGURE 3. EFFECTIVE IMAGE-GATHERING OTF , t(v,w) 





FIGURE 4. IMAGE-GATHERING OPTICS 
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FIGURE 6. OPTIMAL EDGE DETECTOR OTF (c(x,y) = -V L(x,y), 
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FIGURE 6. OPTIMAL EDGE DETECTOR OTF (c(x,y) = -V L(x,y) 
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FIGURE 7. OPTIMAL EDGE DETECTOR PSF (c(x,y) = -V L(x,y) 





FIGURE 7. OPTIMAL EDGE DETECTOR PSF (c(x,y) * -V L(x,y), 





FIGURE 7. OPTIMAL EDGE DETECTOR PSF (c(x,y) » -V L(x,y) 
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(PSF) including the effects of both electrical and optical systems is developed. It is shown that the phase 
distortion appears in the effective image- gathering PSF as a shift in the location of the PSF maximum, an 
asymmetry about the maximum and a decaying “tail” which effectively widens the sensor’s footprint. The 
edges of the acquired image are not only blurred, but also shifted in position along the scan direction. This 
effect further stresses the importance of the combined image gathering and processing approach. It is shown 
that the optimal image processor compensates for both phase and magnitude distortion in the image to the 
extent compatible with noise attenuation, and continues to display a lack of circular symmetry while being 
highly influenced by the sampling lattice used in image acquisition. Finally, it is shown that the spatially 
averaged mean-squared error and its sensitivity to variations in the scene detail and noise statistics are both 
independent of the image-gathering system phase characteristics, when the optimal image processor is used 
to correct magnitude and phase distortion. 
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